reftests: Allow minor differences to be tolerated
authorSimon McVittie <smcv@debian.org>
Sat, 13 Feb 2021 16:19:10 +0000 (16:19 +0000)
committerSimon McVittie <smcv@debian.org>
Tue, 22 Aug 2023 11:11:51 +0000 (12:11 +0100)
commit442ba1f668dff053ddbf9be02ae7cda31e53db44
tree1fa1eff1e750ea6b22db9b42b7964ba92db0206f
parent7b844bc3edc7d8cf4af6668a7d9fed859280bf7e
reftests: Allow minor differences to be tolerated

Based on an earlier patch by Michael Biebl, as used in Debian's GTK 3
packaging, with additional inspiration from librsvg's reftests.

Each .ui or .node reftest can have an accompanying .keyfile file
like this:

    [reftest]
    tolerated-diff-level=20
    tolerated-diff-pixels=1000

If the image differs, but the number of pixels that differ is no more
than tolerated-diff-pixels and the differences are no more than
tolerated-diff-level, then we treat it as a success with warnings, save
the .diff.png for analysis, and use g_test_incomplete() to record the
test-case as "TODO".

Signed-off-by: Simon McVittie <smcv@debian.org>
Forwarded: https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/3195
Applied-upstream: no, upstream want reftests to be a strict pass/fail with identical results required

Gbp-Pq: Topic debian
Gbp-Pq: Name reftests-Allow-minor-differences-to-be-tolerated.patch
testsuite/gsk/compare-render.c
testsuite/reftests/gtk-reftest.c
testsuite/reftests/image-compare.c